.Net: Python: Introducing Release Candidates of Vector Stores and Collections #11482

eavanvalkenburg · 2025-04-10T13:34:22Z

Motivation and Context

Hybrid
Redoes the search mechanisms and adds hybrid search support.
Filters
Adds support for a Callable (lambda func) as a filter

This further does some work on the vector abstractions:

rename is_filterable to is_indexed and is_full_text_searchable to is_full_text_indexed: Closes Port dotnet feature: .Net: Rename IsFilterable to IsIndexed and IsFullTextSearchable to IsFullTextIndexed #11598
remove local_embedding from embedding field, adds has_local_embedding to data field, this allows one to create a data model without local embedding, in the old model that required a vector field anyway, now that can be set on the data field that needs to be embedded by the service.
dimensions is now a required field: Closes Port dotnet feature: .Net: Refactor MEVD Vector Attribute and make Dimensions mandatory #11517
Add support for getting records without a key, with order_by options and top and skip: Closes .Net: Python: Support listing all keys on vector store #9911
Add does_collection_exists and delete_collection methods to VectorStore: Closes Port dotnet feature: .Net: Add delete and exists methods for IVectorStore #11597
Removes the need to import (Vector)SearchOptions, instead uses params and creates the options internal only

Relates to #10561

Description

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

markwallace-microsoft · 2025-04-10T13:37:42Z

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3488	6 💤	0 ❌	0 🔥	1m 42s ⏱️

python/semantic_kernel/connectors/memory/azure_ai_search/azure_ai_search_collection.py

roji · 2025-04-15T14:53:55Z

If I understand correctly, there aren't any integration tests for this, right? I'm seeing what looks like unit tests only for now?

eavanvalkenburg · 2025-04-15T15:00:17Z

@roji indeed int tests still to be done, made a note

Copilot

Pull Request Overview

This PR introduces support for keyword hybrid search and adds lambda filter functionality across multiple memory connectors while updating filter property names. Key changes include:

Adding support for Callable filters (lambda filters) in methods such as _build_filter with corresponding parameter type updates.
Renaming data field properties from is_filterable/is_full_text_searchable to is_indexed/is_full_text_indexed in connectors and samples.
Removing obsolete Azure AI Search connector files to consolidate functionality.

Reviewed Changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
python/semantic_kernel/connectors/memory/pinecone/_pinecone.py	Updated filter handling with lambda support and renaming of filter properties.
python/semantic_kernel/connectors/memory/mongodb_atlas/*	Updated field property checks and filter dictionary creation.
python/semantic_kernel/connectors/memory/in_memory/in_memory_collection.py	Added support for Callable filters and updated error handling for get without keys.
python/semantic_kernel/connectors/memory/chroma/chroma.py	Introduced lambda filter support and updated type imports.
python/semantic_kernel/connectors/memory/azure_cosmos_db/*	Updated filter property usage and query building logic.
python/semantic_kernel/connectors/memory/azure_ai_search/*	Removed obsolete Azure AI Search connector files as part of consolidation.
python/samples/concepts/memory/*	Updated sample data model definitions and search/upsert methods to reflect rename changes.
python/samples/concepts/chat_history/, python/samples/concepts/caching/	Updated field attribute names for filtering and indexing.

Comments suppressed due to low confidence (2)

python/semantic_kernel/connectors/memory/pinecone/_pinecone.py:465

[nitpick] The variable name 'filter' shadows the built-in filter function. Consider renaming it (e.g. 'flt' or 'built_filter') to avoid potential confusion.

if options.filter and (filter := self._build_filter(options.filter)):

python/semantic_kernel/connectors/memory/azure_ai_search/utils.py:1

The entire Azure AI Search connector has been removed. Please ensure that any dependent modules or documentation are updated accordingly and that this change is intentional.

# File removed entirely

python/semantic_kernel/connectors/memory/azure_ai_search.py

python/semantic_kernel/connectors/memory/azure_cosmos_db/azure_cosmos_db_no_sql_collection.py

python/samples/concepts/memory/complex_memory.py

python/semantic_kernel/connectors/memory_stores/azure_cognitive_search/__init__.py

python/semantic_kernel/connectors/memory/pinecone.py

python/semantic_kernel/connectors/memory/azure_ai_search.py

python/semantic_kernel/connectors/search/bing/bing_search.py

python/semantic_kernel/connectors/memory/faiss.py

eavanvalkenburg · 2025-05-16T08:03:07Z

Replace by multiple smaller PRS

eavanvalkenburg requested a review from a team as a code owner April 10, 2025 13:34

markwallace-microsoft added python Pull requests for the Python Semantic Kernel memory labels Apr 10, 2025

eavanvalkenburg force-pushed the hybrid_search branch from e4baa12 to 9e4b565 Compare April 10, 2025 13:34

eavanvalkenburg force-pushed the hybrid_search branch from 590746f to bb5d361 Compare April 15, 2025 08:55

eavanvalkenburg changed the title ~~Python: Introducing Keyword Hybrid Search for Azure AI Search~~ Python: Introducing Keyword Hybrid Search and Lamba Filters for Azure AI Search Apr 15, 2025

eavanvalkenburg changed the title ~~Python: Introducing Keyword Hybrid Search and Lamba Filters for Azure AI Search~~ Python: Introducing Keyword Hybrid Search and Lambda Filters for Azure AI Search Apr 15, 2025

eavanvalkenburg force-pushed the hybrid_search branch from a1924ac to b0f60ae Compare April 15, 2025 14:35

roji reviewed Apr 15, 2025

View reviewed changes

python/semantic_kernel/connectors/memory/azure_ai_search/azure_ai_search_collection.py Outdated Show resolved Hide resolved

eavanvalkenburg force-pushed the hybrid_search branch from a8fc179 to 32aa44c Compare April 16, 2025 08:32

eavanvalkenburg enabled auto-merge April 16, 2025 13:18

eavanvalkenburg force-pushed the hybrid_search branch from b201298 to 0b08684 Compare April 16, 2025 14:33

eavanvalkenburg disabled auto-merge April 16, 2025 19:50

eavanvalkenburg force-pushed the hybrid_search branch 2 times, most recently from 3548400 to 0aa68d5 Compare April 17, 2025 10:17

eavanvalkenburg requested a review from Copilot April 17, 2025 10:17

Copilot AI reviewed Apr 17, 2025

View reviewed changes

eavanvalkenburg force-pushed the hybrid_search branch from 632489d to 4605301 Compare April 17, 2025 12:40

TaoChenOSU reviewed Apr 18, 2025

View reviewed changes

python/semantic_kernel/connectors/memory/azure_ai_search.py Outdated Show resolved Hide resolved

python/semantic_kernel/connectors/memory/azure_cosmos_db/azure_cosmos_db_no_sql_collection.py Outdated Show resolved Hide resolved

eavanvalkenburg force-pushed the hybrid_search branch from 42ede16 to 6e10595 Compare April 29, 2025 15:20

markwallace-microsoft added the documentation label Apr 29, 2025

eavanvalkenburg force-pushed the hybrid_search branch from 14a1bc5 to 3870bc1 Compare April 29, 2025 21:03

moonbox3 reviewed Apr 30, 2025

View reviewed changes

eavanvalkenburg force-pushed the hybrid_search branch 3 times, most recently from c0497d7 to ddccc0a Compare May 1, 2025 20:08

eavanvalkenburg changed the title ~~Python: Introducing Keyword Hybrid Search and Lambda Filters for Azure AI Search~~ Python: Introducing Release Candidates of Vector Stores and Collections May 2, 2025

eavanvalkenburg added 21 commits May 16, 2025 09:15

WIP

9c03b4b

passed mypy

f06a176

fixes

f28f143

updated first batch of tests

b43154a

further tests

669f405

typing

270c296

fixed mypy

2ec7fa9

fixed tests

1dbb313

class to params

e0384e4

updated chroma

440443b

google and brave filters

1a574fe

updated uv

b276e1a

redid vector text search

c9145de

updated hotels sample

79b3c5d

remove file

5da8d05

rebuild lock

a657f02

fixed tests

e4ddb19

fixed mypy

e0c2078

updated azure ai search sample

463f902

fixed py3.13 dep import

95ac65d

added comment

6094b13

eavanvalkenburg force-pushed the hybrid_search branch from 02d8e63 to 6094b13 Compare May 16, 2025 07:17

eavanvalkenburg requested a review from a team as a code owner May 16, 2025 07:17

markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel kernel.core labels May 16, 2025

eavanvalkenburg temporarily deployed to integration May 16, 2025 07:18 — with GitHub Actions Inactive

github-actions bot changed the title ~~Python: Introducing Release Candidates of Vector Stores and Collections~~ .Net: Python: Introducing Release Candidates of Vector Stores and Collections May 16, 2025

eavanvalkenburg closed this May 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

.Net: Python: Introducing Release Candidates of Vector Stores and Collections #11482

.Net: Python: Introducing Release Candidates of Vector Stores and Collections #11482

Uh oh!

eavanvalkenburg commented Apr 10, 2025 •

edited

Loading

Uh oh!

markwallace-microsoft commented Apr 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

roji commented Apr 15, 2025

Uh oh!

eavanvalkenburg commented Apr 15, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eavanvalkenburg commented May 16, 2025

Uh oh!

Uh oh!

.Net: Python: Introducing Release Candidates of Vector Stores and Collections #11482

.Net: Python: Introducing Release Candidates of Vector Stores and Collections #11482

Uh oh!

Conversation

eavanvalkenburg commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Description

Contribution Checklist

Uh oh!

markwallace-microsoft commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python Unit Test Overview

Uh oh!

Uh oh!

roji commented Apr 15, 2025

Uh oh!

eavanvalkenburg commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eavanvalkenburg commented May 16, 2025

Uh oh!

Uh oh!

eavanvalkenburg commented Apr 10, 2025 •

edited

Loading

markwallace-microsoft commented Apr 10, 2025 •

edited

Loading

eavanvalkenburg commented Apr 15, 2025 •

edited

Loading